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Abstract 



^~~i This paper studies the deviations of the regret in a stochastic muhi-armed bandit 

l/^ problem. When the total number of plays n is known beforehand by the agent, 

OQ Audibert et al. [2] exhibit a policy such that with probability at least 1 — 1/n, 

^^ the regret of the policy is of order logn. They have also shown that such a 

^ property is not shared by the popular UCBI policy of Auer et al. [3]. This work 

^H first answers an open question: it extends this negative result to any anytime 

^H policy. Another contribution of this paper is to design anytime robust policies 

for specific multi-armed bandit problems in which some restrictions are put on 

the set of possible distributions of the different arms. We also show that, for 

any policy (i.e. when the number of plays is known), the regret is of order logn 

with probability at least 1 — 1/n, so that the policy of Audibert et al. has the 

^N best possible deviation properties. 

> 
^>0 Keywords: exploration-exploitation tradeoff, multi- armed stochastic bandit, 

^— ' regret deviations /risk 

in 



1. Introduction 



,__! Bandit problems illustrate the fundamental difficulty of sequential decision 

• • making in the face of uncertainty: a decision maker must choose between fol- 

. J^ lowing what seems to be the best choice in view of the past ( "exploitation" ) or 

S^ testing ( "exploration" ) some alternative, hoping to discover a choice that beats 

^ the current empirical best choice. More precisely, in the stochastic multi-armed 

bandit problem, at each stage, an agent (or decision maker) chooses one action 
(or arm), and receives a reward from it. The agent aims at maximizing his 
rewards. Since he does not know the process generating the rewards, he does 
not know the best arm, that is the one having the highest expected reward. He 
thus incurs a regret, that is the difference between the cumulative reward he 
would have got by always drawing the best arm and the cumulative reward he 
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actually got. The name "bandit" comes from imagining a gambler in a casino 
playing with K slot machines, where at each round, the gambler pulls the arm 
of any of the machines and gets a payoff as a result. 

The multi-armed bandit problem is the simplest setting where one encoun- 
ters the exploration-exploitation dilemma. It has a wide range of applications 
including advertisement [4, 9], economics [5, 18], games [11] and optimization 
[15, 8, 14, 6]. It can be a central building block of larger systems, like in evolu- 
tionary programming [12] and reinforcement learning [23], in particular in large 
state space Markovian Decision Problems [16]. Most of these applications re- 
quire that the policy of the forecaster works well for any time. For instance, in 
tree search using bandit policies at each node, the number of times the bandit 
policy will be applied at each node is not known beforehand (except for the root 
node in some cases) , and the bandit policy should thus provide consistently low 
regret whatever the total number of rounds is. 

Most previous works on the stochastic multi-armed bandit [21, 17, 1, 3, 
among others] focused on the expected regret, and showed that after n rounds, 
the expected regret is of order log n. So far, the analysis of the upper tail of 
the regret was only addressed in Audibert ct al. [2] . The two main results there 
about the deviations of the regret are the following. First, after n rounds, for 
large enough constant C > 0, the probability that the regret of UCBl (and also 
its variant taking into account the empirical variance) exceeds C log n is upper 
bounded by l/(logn)'-^ for some constant C" depending on the distributions of 
the arms and on C (but not on n). Besides, for most bandit problems, this 
upper bound is tight to the extent that the probability is also lower bounded by 
a quantity of the same form. Second, a new upper confidence bound policy was 
proposed: it requires to know the total number of rounds in advance and uses 
this knowledge to design a policy which essentially explores in the first rounds 
and then exploits the information gathered in the exploration phase. Its regret 
has the advantage of being more concentrated to the extent that with probability 
at least 1 — 1/n, the regret is of order logn. The problem left open by [2] is 
whether it is possible to design an anytime robust policy, that is a policy such 
that for any n, with probability at least 1 — 1/n, the regret is of order logn. In 
this paper, we answer negatively to this question when the reward distributions 
of all arms are just assumed to be uniformly bounded, say all rewards arc in 
[0, 1] for instance (Corollary 3.4). We then study which kind of restrictions on 
the set of probabilities defining the bandit problem allows to answer positively. 
One of our positive results is the following: if the agent knows the value of the 
expected reward of the best arm (but does not know which arm is the best one), 
the agent can use this information to design an anytime robust policy (Theorem 
4.3). We also show that it is not possible to design a policy such that the regret 
is of order logn with a probability that would significantly greater than 1 — 1/n, 
even if the agent knows the total number of rounds in advance (Corollary 5.2). 
The paper is organised as follows: in the first section, we formally describe 
the problem we address and give the corresponding definitions and properties. 
Next we present our main impossibility result. In the third section, we provide 
restrictions under which it is possible to design anytime robust policies. In the 



fourth section, we study the robustness of pohcies that can use the knowledge 
of the total number of rounds. Then we provide experiments to compare our 
robust policy to the classical UCB algorithms. The last section is devoted to 
the proofs of our results. 

2. Problem setup and definitions 

In the stochastic multi-armed bandit problem with K > 2 arms, at each 
time step t = 1,2,..., an agent has to choose an arm It in the set {1, . . . , K^ 
and obtains a reward drawn from vi^ independently from the past (actions 
and observations). The environment is thus parameterized by a K-ixvple of 
probability distributions = {vi, . . . , vk)- The agent aims at maximizing his 
rewards. He does not know but knows that it belongs to some set Q. We 
assume for simplicity that O C 0, where G) denotes the set of all iiT-tuple of 
probability distributions on [0, 1]. We thus assume that the rewards are in [0, 1]. 

For each arm k and all times t > 1, let Tk{t) — X]s=i '^is=k denote the num- 
ber of times arm k was pulled from round 1 to round t, and X}^i, X]^2^ ■ ■ ■ i ^fc,Tfc(t) 
the sequence of associated rewards. For an environment parameterized by 
— (i^i, . . . , Vk), let Pe denote the distribution on the probability space such 
that for any k € {1, . . . , K}, the random variables Xk^i, Xk^2, ■ ■ ■ are i.i.d. real- 
izations of Vk, and such that these K infinite sequence of random variables are 
independent. Let Eg denote the associated expectation. 

Let Hk = j xdvk{x) be the mean reward of arm k. Introduce jjl* — maxfc ^k 
and fix an arm k* G argmaxj.gM ^^i /i^, that is k* has the best expected 
reward. The suboptimality of arm k is measured by A^; — ^* — fJ-k- The agent 
aims at minimizing its regret defined as the difference between the cumulative 
reward he would have got by always drawing the best arm and the cumulative 
reward he actually got. At time n > 1, its regret is thus 

n n 

Rn^^Xk'^t-^Xj^^Ti^it)- (1) 

t=l t=l 

The expectation of this regret has a simple expression in terms of the subop- 
timalities of the arms and the expected sampling times of the arms at time n. 
Precisely, we have 

n , K 

EgEn = n^* - y^ Eg (yU/^ ) = n[i* - Eg f y^ Tfc [n)[ik 

K K K 

= [i*^Ee [Tfc (n)] - ^ A^feEe [T^ (n)] = ^ AfeEg [T^ (n)] . 

k=\ k = \ k=\ 

Other notions of regret exists in the literature: the quantity ^Ylik=\ ^kTk{n) is 
called the pseudo regret and may be more practical to study, and the quantity 
maxfc X]r=i -^k,t ~ X]"=i -^it-Ti (t) defines the regret in adverserial settings. Re- 
sults and ideas we want to convey here are more suited to definition (1), and 



taking another definition of the regret would only bring some more technical 
intricacies. 

Our main interest is the study of the deviations of the regret Rn, i.e. the 
value of Pg{Rn > x) when x is larger and of order of WlgRn. If a policy has 
small deviations, it means that the regret is small with high probability and in 
particular, if the policy is used on some real data, it is very likely to be small 
on this specific dataset. Naturally, small deviations imply small expected regret 
since we have 

EgRn < Eg max(7?„, 0) = / Vg (kn > x) dx. 

To a lesser extent it is also interesting to study the deviations of the sampling 
times Tn{k), as this shows the ability of a policy to match the best arm. More- 
over our analysis is mostly based on results on the deviations of the sampling 
times, which then enables to derive results on the regret. We thus define below 
the notion of being /-upper tailed for both quantities. 

Define M^ = {x e M : x > 0}, and let A — min^^fc. A^ be the gap between the 
best arm and second best arm. 

Definition 1 (/-T and f-TZ). Consider a mapping / : K — > K;^. A policy has 
f -upper tailed sampling Times (in short, we will say that the policy is f-T) if 
and only if 



3C, C > 0, V6' e e such that A 7^ 0, 



Vn > 2, Vfc ^ k*, ¥g ( Tkin) > C^ ] < 



logn\ ^ C 
^l J - fin) 



A policy has f -upper tailed Regret (in short, f-TZ) if and only if 

3C, C>0, Ve e e such that A 7^ 0, Vn > 2, ¥g ( i?„ > C^^ ) < '^ 



A ; - fin)- 

We will sometimes prefer to denote fin)-T (resp. f{n)-TZ) instead of f-T 
(resp. f-TZ) for readability. Note also that, for sake of simplicity, we leave aside 
the degenerated case of A being null (i.e. when there are at least two optimal 
arms) . 

In this definition, we considered that the number K of arms is fixed, meaning 
that C and C may depend on K. The thresholds considered on T]^{n) and 
Rn directly come from known tight upper bounds on the expectation of these 
quantities for several policies. To illustrate this, let us recall the definition and 
properties of the popular UCBI policy. Let Xk^s = - X]t=i -^k.t be the empirical 
mean of arm k after s pulls. In ucbI, the agent plays each arm once, and then 
(from f > ii' + 1), he plays 




J4 e argmax < ^fc,T^,(t-i) + J 7^777 — z-,)- (2) 

fce{i,...,-ff} I 



While the first term in the bracket ensures the exploitation of the knowledge 
gathered during steps 1 to i — 1, the second one ensures the exploration of the 
less sampled arms. For this policy, Auer et al. [3] proved: 

Vn>3, E[rfe(n)] < 12^ and Eei?„ < 12^^ ^ < 12if ^. 

Lai and Robbins [17] showed that these results cannot be improved up to numer- 
ical constants. Audibert et al. [2] proved that ucbI is log -T and log' -TZ where 
log is the function x i— ?• [log(a;)]'^. Besides, they also study the case when 2\ogt 
is replaced by plogt in (2) with p > 0, and proved that this modified ucbI 
is log^''^^-T and log^''^^-7?. for p > 1/2, and that p = | is actually a critical 
value. Indeed, for p < 1/2 the policy does not even have a logarithmic regret 
guarantee in expectation. Another variant of UCBl proposed by Audibert et al. 
is to replace 21ogt by 21ogn in (2) when we want to have low and concentrated 
regret at a fixed given time n. We refer to it as ucb-h as its implementation 
requires the knowledge of the horizon n of the game. The behaviour of ucb-h on 
the time interval [1, n] is significantly different to the one of UCBI, as ucb-h will 
explore much more at the beginning of the interval, and thus avoids exploiting 
the suboptimal arms on the early rounds. Audibert et al. showed that ucb-h is 
n-T and n-TZ (as it will be recalled in Theorem 3.5). As it will be confirmed by 
our results, whether a policy knows in advance the horizon n or not matters a 
lot, that is why we introduce the following terms. 

Definition 2. A policy that uses the knowledge of the horizon n (e.g. UCB-HJ 
is a horizon policy. A policy that does not use the knowledge of n (e.g. UCBlj 
is an anytime policy. 

We now introduce the weak notion of /-upper tailed as this notion will be 
used to get our strongest impossibility results. 

Definition 3 (/-wT and f-wTZ). Consider a mapping / : M — > M^. A policy 
has weak f -upper tailed sampling Times (in short, we will say that the policy is 
f-wT) if and only if 



V6' e e such that A 7^ 0, 

Mn \ _C_ 
^l J - fin) 
A policy has weak f -upper tailed Regret (in short, f-wTZ) if and only if 



3C, C > 0, Vn > 2, \/k^k\ Ve[ Tk{n) > C^^ ] < 



Wee such that A 7^ 0, 3C,C> 0, Vn > 2, P^ ( i?„ > C^^^] < '^ 



A y - fin) 

The only difference between f-T and f-wT (and between f-TZ and f-wTZ) 
is the interchange of "V0" and "3C, C"' . Consequently, a policy that is f-T 
(respectively f-TZ) is f-T (respectively f-wTZ). Let us detail the links between 
the f-T, f-TZ, f-wT and f-wTZ. 



Proposition 2.1. Assume that there exists a, (3 > such that f{n) < an^ for 
any n > 2. We have 

f-T => f-n => f-wU ^ f-wT. 

The proof of this proposition is technical but rather straightforward. Note 
that we do not have f-TZ =^ f-T, because the agent may not regret having 
pulled a suboptimal arm if the latter has delivered good rewards. Note also 
that / is required to be at most polynomial: if not some rare events such as 
unlikely deviations of rewards towards their actual mean can not be neglected, 
and none of the implications hold in general (except, of course, f-TZ => f-wTZ 
and f-T ^ f-wT). 

3. Impossibility result 

Here and in section 4 we mostly deal with anytime policies, and the word 
policy (or algorithm) implicitly refers to anytime policy. 

In the previous section, we have mentioned that for any p > 1/2, there is 
a variant of UCBl (obtained by changing 21ogi into plogt in (2)) which is 
log ''^ -T. This means that, for any a > 0, there exists a log"-T policy, and a 
hence log"-??, policy. The following result shows that it is impossible to find an 
algorithm that would have better deviation properties than these UCB policies. 
For many usual settings (e.g., when Q is the set €) of all iiT-tuples of measures on 
[0, 1]), with not so small probability, the agent gets stuck drawing a suboptimal 
arm he believes best. Precisely, this situation arises when simultaneously: 

(a) an arm k delivers payoffs according to a same distribution i/k in two dis- 
tinct environments 9 and 9, 

(b) arm k is optimal in 9 but suboptimal in 9, 

(c) in environment 9, other arms may behave as in environment 9, i.e. with 
positive probability other arms deliver payoffs that are likely in both en- 
vironments. 

If the agent suspects that arm k delivers payoffs according to Vk, he does not 
know if he has to pull arm k again (in case the environment is 9) or to pull the 
optimal arm of 9. The other arms can help to point out the difference between 
9 and 9, but then they have to be chosen often enough. This is in fact this kind 
of situation that has to be taken into account when balancing a policy between 
exploitation and exploration. 

Our main result is the formalization of the leads given above. In particular, 
we give a rigorous description of conditions (a) , (b) and (c) . Let us first recall 
the following results, which are needed in the formalization of condition (c). 
One may look at [22], p. 121 for details (among others). Those who are not 
familiar with measure theory can skip to the non- formal explanation just after 
the results. 



Theorem 3.1 (Lebesgue-Radon-Nikodym theorem). Let jii and /i2 be a-finite 
measures on a given measurable space. There exists a ij,2-integrable function -^ 
and a a-finite measure m such that m and fj,2 are singular"^ and 

dfii 
Ml = 1 — ■ ti'2 + m. 

The density -4^ is unique up to a ^2-negligible event. 



dfj.2 

dfj,2 



We adopt the convention that -J^ = +oo on the complementary of the 



support of M2- 
Lemma 3.2. We have 
•Mi(ti=0)=0. 

Proof. The first point is a clear consequence of the decomposition /ii = -j^ ■ 
/Z2 + TO and of the convention mentioned above. For the second point, one can 
write by uniqueness of the decomposition: 

/ drii \ dill 

/i2 -; — > = ^ - — = /i2 — a.s. <^ /ii = m -^ fii and fi2 are smgular. 

V "M2 / «M2 

And by symmetry of the roles of fii and 112'- 

M2 I -; — > I > <=> Ml ^nd ^2 are not singular <=> fii I - — > ) > 0. 
\d^2 J V"Mi / 

D 

Let us explain what these results have to do with condition (c). 
One may be able to distinguish environment 9 from if a certain arm £ delivers 
a payoff that is infinitely more likely in 6 than in 9. This is for instance the case 
if Xi^t is in the support of Dg and not in the support of i^i, but our condition is 
more general. If the agent observes a payoff x from arm £, the quantity ^ (x) 
represents how much the observation of x is more likely in environment 9 than in 
9. If Vk and i>k admit density functions (say, respectively, / and /) with respect 
to a common measure, then -^(x) — tt^- Thus the agent will almost never 

' ai/i ^ ' f{x) ° 

make a mistake if he removes 9 from possible environments when -p^{x) = 0. 
This may happen even if x is in both supports of i>i and i>^, for example if x is 
an atom of Og and not of V£ (i.e. i^eix) > and h'i{x)—0). On the contrary, if 
-j|^(x) > both environments 9 and 9 are likely and arm ^'s behaviour is both 



^Two measures mi and 1712 on a measurable space (Q, J-') are singular if and only if there 
exists two disjoint measurable sets Ai and A2 such that Ai U A2 = f2, mi(A2) = and 
m2{Ai) = 0. 



consistent with 6 and 6. 

Now let us state the impossibility result. Here and throughout the paper 
we find it more convenient to denote / 3>+oo 9 rather than the usual notation 
g = o{f), which has the following meaning: 

Ve > 0, 3N> 0, Vn > N, g{n) < ef{n). 

Theorem 3.3. Let / : N ^ M^ he greater than order log", that is for any 

a>0, />+oo log". 

Assume that there exists 9, 9 (z Q, and k G {1, . . . , K} such that: 

(a) i^k^h, 

(h) k is the index of the best arm in 9 but not in 9 , 

(e)yi^k, P,^(^(X,,i)>0)>0. 

Then there is no f-wT anytime policy, and hence no f-TZ anytime policy. 

Let us give some hints of the proof (see Section 7 for details). The main 
idea is to consider a policy that would be /-wT, and in particular that would 
"work well" in environment 9 in the sense given by the definition of f-wT. The 
proof exhibits a time N at which arm k, optimal in environment 9 and thus 
often drawn with high Pg-probability, is drawn too many times (more than the 
logarithmic threshold C °^2 ) with not so small Pg-probability, which shows 

k 

the nonexistence of such a policy. More precisely, let n be large enough and 
consider a time A^ of order logn and above the threshold. If the policy is /-wT, 
at time N, sampling times of suboptimal arms arc of order log A^ at most, with 
Pe -probability at least 1 — C/f(N). In this case, at time A^, the draws are con- 
centrated on arm k. So Tk{N) is of order N, which is more than the threshold. 
This event holds with high Pg-probability. Now, from (a) and (c), we exhibit 
constants that are characteristic of the ability of arms £ y^ k to "behave as if in 
0": for some < a, 77 < 1, there is a subset ^ of this event such that Pe(^) > a^ 
for T = '^i^]^Ti{N) and for which ^ is lower bounded by rj^ . The event ^ 
on which the arm k is sampled N times at least has therefore a Pg-probability 
of order (rja)^ at least. This concludes this sketchy proof since T is of order 
logiV, thus (rja)^ is of order log°^^''''' n at least. 

Note that the conditions given in Theorem 3.3 are not very restrictive. The 
impossibility holds for very basic settings, and may hold even if the agent has 
great knowledge of the possible environments. For instance, the setting 

if = 2ande = |(ser(l),J.),(i?er(^),<5. 

where Ber{p) denotes the Bernoulli distribution of parameter p and Sx the Dirac 
measure on x, satisfies the three conditions of the theorem. 
Nevertheless, the main interest of the result regarding the previous literature is 
the following corollary. 



Corollary 3.4. IfQ is the whole set Q of all K -tuples of measures on [0, 1], then 
there is no f-TZ anytime policy, where f is any function such that f ^+oo log" 
for all a > 0. 

This corollary should be read in conjunction with the following result for 
UCB-H which, for a given n, plays at time t > K + 1, 



<. / 21ogn 

It G argmax <^ Xfe,T,(t-i) + \ Tfrji — tt >■ 
ke{i,...,K} [ V Jfe(t- ij J 

Theorem 3.5. For any /? > 0, ucb-h is n^-TZ. 

For /9 > 1, Theorem 3.5 can easily be extended to the policy UCB-H(p) which 
starts by drawing each arm once, and then at time t> K -\-l, plays 



It e argmax \Xk,T^{t-i) + \\ J! ,, \. \- (3) 

Naturally, we have n^ ^„_).+oo log"(n) for all a,/3 > but this does not 
contradict our theorem, since UCB-H(p) is not an anytime policy, ucb-h will 
work fine if the horizon n is known in advance, but may perform poorly at other 
rounds. 

Corollary 3.4 should also be read in conjunction with the following result for 
the policy UCBl(p) which starts by drawing each arm once, and then at time 
t>K + l, plays 



JtG argmax <i ^/c,T.(t-i) + W 77^7^^^ ^ (4) 

ke{i,...,K} y V Jfe(r- Ij J 

Theorem 3.6. For any p > 1/2, \}C^al(p) is \og^''^^-Tl. 

Thus, any improvements of existing algorithms which would for instance 
involve estimations of variance (see [2]), of A^,, or of many characteristics of the 
distributions cannot beat the variants of UCbI regarding deviations. 

4. Positive results 

The intuition behind Theorem 3.3 suggests that, if one of the three condi- 
tions (a), (b), (c) does not hold, a robust policy would consist in the following: 
at each round and for each arm fc, compute a distance between the empirical 
distribution of arm k and the set of distribution i^j, that makes arm k optimal 
in a given environment 9. As this distance decreases with our belief that k is 
the optimal arm, the policy consists in taking the k minimizing the distance. 
Thus, the agent chooses an arm that fits better a winning distribution Vk- He 
cannot get stuck pulling a suboptimal arm because there are no environments 



9 with i^k ~ Vk in which k would be suboptimal. More precisely, if there exists 
such an environment 9, the agent is able to distinguish 9 from 0: during the 
first rounds, he pulls every arm and at least one of them will never behave as if 
in 9 if the current environment is 9. Thus, in 9, he is able to remove 9 from the 
set of possible environments O (remember that is a parameter of the problem 
which is known by the agent). 

Nevertheless such a policy cannot work in general, notably because of the 
three following limitations: 

• If is the current environment and even if the agent has identified 9 as 
impossible (i.e. ^^{Xk^i) ~ 0), there still could be other environments 9' 
that are arbitrary close to 9 in which arm k is optimal and which the agent 
is not able to distinguish from 9. This means that the agent may pull arm 
k too often because distribution t'j, — v^. is too close to a distribution i^^ 
that makes arm k the optimal arm. 

• The ability to identify environments as impossible relies on the fact that 
the event j~^iXk^i) > is almost sure under Pg (see Lemma 3.2). If 
the set of all environments O is uncountable, such a criterion can lead 
to exclude the actual environment. For instance, assume an agent has to 
distinguish a distribution among all Dirac measures Sx {x € [0,1]) and 
the uniform probability A over [0, 1]. Whatever the payoff x observed by 
the agent, he will always exclude A from the possible distributions, as x is 
always infinitely more likely under Sx than under A: 

Va;€[0,l], ^(^)=0. 

• On the other hand, the agent could legitimately consider an environment 9 
as unlikely if, for £ > small enough, there exists 9 such that ^(X^ i) < 
s. Criterion (c) only considers as unlikely an environment 9 when there 
exists 9 such that '^{XkA) = 0. 

Despite these limitations, we give in this section sufficient conditions on O 
for such a policy to be robust. This is equivalent to finding conditions on O 
under which the converse of Theorem 3.3 holds, i.e. under which the fact one 
of the conditions (a), (b) or (c) does not hold implies the existence of a robust 
policy. This can also be expressed as finding which kind of knowledge of the 
environment enables to design anytime robust policies. 

We estimate distributions of each arm by means of their empirical cumulative 
distribution functions, and distance between two c.d.f. is measured by the norm 
||.||oo, defined by ||/||oo = sup^g[o,i] 1/(^)1 where / is any function [0,1] -^ JR. 
The empirical c.d.f of arm k after having been pulled t times is denoted Fk^t- 
The way we choose an arm at each round is based on confidence areas around 
Fk,Tk{n-i)- We choose the greater confidence level (gcl) such that there is still 
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Proceed as follows: 




• Draw each arm once. 




• Remove each 9 £ Q such that there exists ^ G 6 and £ G {1, • • 


. , K} with 


• Then at each round t, play an arm 




It G argmin Tkit - 1) irif ffe,Tfc(t-i) - F^^ f 
ke{i,...,K} ^ec-jfc 





Figure 1; A c.d.f.-based algorithm: GCL. 

an arm k and a winning distribution i/k such that i^^^., the c.d.f. of i>k, is in 
the area of F}^Tk{n-i)- We then select the corresponding arm k. By means of 
Massart's inequality (1990), this leads to the c.d.f. based algorithm described 
in Figure 1. Q^ denotes the set {6' G Q\k is the optimal arm in 6'}, i.e. the set 
of environments that makes k the index of the optimal arm. 

^.1. Q is finite 

When O is finite the limitations presented above do not really matter, so 
that the converse of Theorem 3.3 is true and our algorithm is robust. 

Theorem 4.1. Assume that O is finite and that for all 9 — (i^i, . . . , i^/f), ^ = 
(f'l, . . . , i>k) G O, and all k G {1, . . . , K}, at least one of the following holds: 

• k is suboptimal in 9 , or is optimal in 9. 

.3£^k, P^^(^(Xo)>0)^0. 

Then GCL is n^ -T (and hence n^-TZ) for all 13 > 0. 

4-2. Bernoulli laws 

We assume that any Vk {k G {1, . . . ,K}, 9 G Q) is a Bernoulli law, and 
denote by Hk its parameter. We also assume that there exists 7 G (0, 1) such that 
/Xfc G [7, 1] for all k and all 9.^ Moreover we may denote arbitrary environments 
9,9 hy 9 = (mi,... ,^j.k) and 9 = {fii, . . . ^fix)- 

In this case ^^{l) ^ ^^ > Q, so that for any 9, 9 e B and any ? G {1, . . . , K} 
one has 

^(X,a) >o]> P,-(X,,i = l) = fii>0. 

Therefore condition (c) of Theorem 3.3 holds, and the impossibility result only 
relies on conditions (a) and (b). Our algorithm can be made simpler: there is no 



^The result also holds if all parameters /ife are in a given interval [0, 7], 7 G (0, 1). 
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Proceed as follows: 




• Draw each arm once. 




• Then at each round t, play an arm 




It G argmin Tk(t — 1) inf (fit, - 
fc6{i,...,K} eeSfc V 


^^fe,Tfc(t-l)j • 



Figure 2; A c.d.f.-bascd algorithm in case of Bernoulli laws: GCL-B. 

need to try to exclude unlikely environments, and computing the empirical c.d.f. 
is equivalent to computing the empirical mean (see Figure 2). The theorem and 
its converse are expressed as follows. We will refer to our policy as GCL-b as it 
looks for the environment matching the observations at the Greatest Confidence 
Level, in the case of Bernoulli distributions. 

Theorem 4.2. For any 9 E and any k E {I, . . . , K}, let us set 

dk = inf \Hk - Afel- 
eeSk 



GCL-B is such that: 

V/3>0, 3C,C>O,V0Ge, Vn> 1, Vfc € {1, . . . ,X}, ¥g [Tk{n) > '" g"'" 1 < ^. 



Clogn\ C 



Let / : N* ^ K+ be greater than order log".- Va > 0, / >+oo log". 
// there exists k such that 

(a') inf dk = inf |^fe - /i^l = 0, 
0ee\efc « e e^ 

then there is no anytime policy such that: 

3C,C>0y9ee, Vn>2, \/k^k*, f>e{Tk(n)>C\ogn) < -^. 

Note that we do not adopt the former definitions of robustness {f-TZ and 
f-T), because the significant term here is d^ (and not A^)'^, which represents 
the distance between Ok and Q \ Ok- Indeed robustness lies on the ability 
to distinguish environments, and this ability is all the more stronger as the 
distance between the parameters of these environments is greater. Provided 
that the density ji is uniformly bounded away from zero, the theorem holds 
for any parametric model, with dk being defined with a norm on the space of 



•^There is no need to leave aside the case of dj. = 0: with the convention g = +oo, the 
corresponding event has zero probability. 
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Proceed as follows: 

• Draw each arm once. 

• Then at each round t, play an arm 

, 2 



It G argmin Tk{t - 1) (^* - X^^T^Ct-i) 

ke{l,...,K} ^ 



Figure 3: GCL*: a variant of c.d.f.-bascd algorithm when fi* is known. 

parameters (instead of |.|). 

Note also that the second part of the theorem is a bit weaker than Theorem 
3.3, because of the interchange of "V0" and "3C, (7" . The reason for this is that 
condition (a) is replaced by a weaker assumption: i^k does not equal Vk, but 
condition (a') means that such Vk and % can be chosen arbitrarily close. 

4-. 3. fi* is known 

This section shows that the impossibility result also breaks down if /i* is 
known by the agent. This situation is formalized as /i* being constant over O. 
Conditions (a) and (b) of Theorem 3.3 do not hold: if a distribution j/j, makes 
arm k optimal in an environment 0, it is still optimal in any environment 9 such 
that Vk = Vk- 

In this case, our algorithm can be made simpler (see Figure 3). At each round 
we choose the greatest confidence level such that at least one empirical mean 
Xk^Tk(t-i) has /i* in its confidence interval, and select the corresponding arm 
k. This is similar to the previous algorithm, deviations being evaluated ac- 
cording to Hoeffding's inequality instead of Massart's one. There is one more 
refinement: the level confidence of arm k at time step t can be defined as 
Tk{t — l)(/i* — Xk^Tk{t-i))+ (where, for any a; G M, X-|_ denotes max(0,a;)) in- 
stead of Tk{t— l)(/x* — -^fe,Tfc(t-i))^- Indeed, there is no need to penalize an 
arm for his empirical mean reward being too much greater than fi* . We will 
refer to this policy as GCL*. 

Theorem 4.3. When /i* is known, GCL* is n^-T (and hence n^ -R) for all 
/3>0. 

GCL* relies on the use of Hoeffding's inequality. It is now well-established 
that in general, the Hocffding inequality does not lead to the best factor in 
front of the log n in the expected regret bound. The minimax factor has been 
identified in the works of Lai and Robbins [17], Burnetas and Katehakis [7] for 
specific families of probability distributions. This result has been strengthened 
in Honda and Takemura [13] to deal with the whole set of probability distribu- 
tions on [0, 1]. Getting the best factor in front of the logn term in the expected 
regret bound appeared there to be tightly linked with the use of Sanov's in- 
equality. The recent work of Maillard et al. [19] builds on a non-asymptotic 
version of Sanov's inequality to get tight non-asymptotic bounds for probability 
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distributions with finite support. Garivicr and Cappe [10] adopts a different 
starting point: tlie Cfiernoff inequaiity. This inequaiity states that for i.i.d. 
random variabies V,Vi, . . . ,Vt, talcing their vaiues in [0, 1], for any r < EF we 
have 



i=i ' 



<exp(-r/C(T,EF)), (5) 



where /C(p, q) denotes the Kuilbaclc-Leibier divergence between Bernouiii distri- 
butions of respective parameter p and q. It is known to be tight for Bernouiii 
random variabies (as discussed e.g. in [10]). A Chernoff version of GCL* would 
consist in the following: 

/t e argmin Tfe(t- 1)/C( min(Xfe_Tfc(t-i),M*) 7 A^*)- (6) 

k(i{\,...,K} 

At the expense of a more refined analysis, it is easy to prove that Theorem 4.3 
still holds for this algorithm. Getting a small constant in front of the logarithmic 
term being an orthogonal discussion to the main topic of this paper, we do not 
detail further this point. 

5. Horizon policies 

We now study regret deviation properties of horizon policies. Again, we 
prove that UCB policies are optimal. Indeed, deviations of ucb-h are of order 
1/n" (for all a > 0) and our result shows that this cannot be improved in 
general. 

This second impossibility result holds for many settings, that is the one for 
which there exists 8,9 E Q such that: 

(b) an arm k is optimal in 9 but not in 9, 

(c') in environment 9, all arms may behave as if in 9. 

Indeed, draws have to be concentrated on arm k in environment 9. In partic- 
ular, with large Pg-probability, the number of draws of arm k (and only of arm 



T^ c log " 



Such an 



k) exceed the logarithmic threshold ^f " at step N 

event only affects a small (logarithmic) number of pulls, so that in environment 
9 arms may easily behave as in 9, and arm k is pulled too often with not so small 
Pg-probability. More precisely, this event happens with at least Pg-probability 

1 — ,,(^ for a /-wT policy. Because arms under environment 9 are able to 
behave as in 9, there exist constants < a, 77 < 1 and a subset £, of this event 
such that Pe('C) ^ o^ and for which ^pr is lower bounded by rj^ . The event 
^ has then Pg-probability of order (rja)^ at least. As N is of order logn, the 
probability of arm k being pulled too often in 9 is therefore at least of order l/n 
to the power of a constant. Hence the following result. 

Theorem 5.1. Let / : N — > M^ be greater than order n" , that is for any 

a> 0, f{n) >„^+co J^"- 

Assume that there exists 9 , 9 E Q, and k G {1, . . . , K} such that: 
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(b) k is the index of the best arm in 9 but not in 6 , 

(e')yie{l,...,K}, P,-(^(X,,i)>0)>0. 

Then there is no f-wT horizon policy, and hence no f-T horizon policy. 

Note that the conditions under which the impossibihty holds are far less 
restrictive than in Theorem 3.3. Indeed, conditions (b) and (c') are equivalent 
to: 

(b) k is the index of the best arm in 9 but not in 9, 

(c) V£^fc, P,~(^(X,,i)>0)>0. 

These are the same conditions as in Theorem 3.3, except for the first one, 
(a"), which is weaker than condition (a). 

As a consequence, corollary 3.4 can also be written in the context of horizon 
policies. 

Corollary 5.2. IfQ is the whole set Q of all K -tuples of measures on [0, 1], then 
there is no f-T horizon policy, where f is any function such that f{n) 3>rn.+oo 
n" for all a > 0. 

Moreover, the impossibility also holds for many basic settings, such as the 
ones described in section 4.2. This shows that GCL-b is not only better in terms 
of deviations than UCB anytime algorithms, but it is also optimal and, despite 
being an anytime policy , it is at least as good as any horizon policy. In fact, 
in most settings suitable for a GCL algorithm, GCL is optimal and is as good as 
UCB-H without using the knowledge of the horizon n. 

Nevertheless, the impossibility is not strong enough to avoid the existence 
of f-TZ horizon policies, with f{n) ^„^+oo n°' and any a > 0. Proposition 
2.1 does not enable to deduce the non-existence of f-TZ policy from the non- 
existence of /-wT policy because it needs / to be less than a function of the form 
an^ . We believe that, in general, the impossibility still holds for f-TZ horizon 
policies, but the corresponding conditions will not be easy to write and the 
analysis will not be as clear as our previous results. Basically, the impossibility 
would require the existence of a pair of environments 9, 9 such that 

• an arm k is optimal in 9 but not in 9, 

• in environment 9, all arms may behave as if in 9 in such a way that best 
arm in would have actually given greater rewards than the other arms 
if it had been pulled more often. 

Finally, as in section 4 one can wonder if there exists a converse to our 
result. Again, such an analysis would be tougher to perform and we only give 
some basic hints. 
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If Q is such that for any 9,0 G & either (b) or (c') does not hold, then one could 
actually perform very well. In this degenerated case, only one pull of each arm 
may make it possible to distinguish tricky pairs of environments 9,6, and thus 
to learn the best arm k* . The agent then keeps on pulling arm k* , and its regret 
is almost surely less than K at any time step. The tricky part is that, as the 
distinction relies on the fact that the event j^{Xk^i) > is almost sure under 
Pe (see Lemma 3.2), this may not work if O is uncountable. 

6. Experiments 

Our goal is to compare anytime UCB policies, more precisely UCBl(p) for 
p > defined by (4), to the low-deviation policies UCB-h(/9) for p > 0, defined 
by (3), and GCL* introduced in Section 4 (see Figure 3). Most bandit policies 
contain a parameter allowing to tunc the exploration-exploitation trade-off. To 
do a fair comparison with anytime UCB policies, we consider the full range of 
possible exploration parameters. 

We estimate the distribution of the regret of a policy by running 100000 sim- 
ulations. In particular, this implies that the confidence interval for the expected 
regret of a policy is smaller than the size of the markers for n — 100, and smaller 
than the lincwidth for n > 500. The reward distribution of the arms are here 
either the uniform (Unif), or the Bernoulli (Ber) or the Dirac distributions. 

6.1. Tuning the exploration parameter in uCB policies for low expected regret 

In UCB policies, the exploration parameter can be interpreted as the confi- 
dence level at which a deviation inequality is applied (neglecting union bounds 
issues). For instance, the popular ucbI uses an exploration term y/ {2 log t)/Ti^{t) 
y^iog(Fy7(22Y(^ corresponding to a 1/t^ confidence level in view of Hoeffd- 
ing's inequality. Several studies [17, 1, 7, 2, 13] have shown that the critical 
confidence level is l/t. In particular, [2] have considered the policy ucb1(/9) 
having the exploration term y^ {p log t)/Tk{t), and shown that this policy have 
polynomial regrets as soon as p < 1/2 (and have logarithmic regret for p > 1/2). 
Precisely, for p < 1/2, the regret of the policy can be lower bounded by n'^ with 
< 7 < 1 which is all the smaller as p is close to 1/2. 

The first experiments, reported in Figures 4 and 5, show that for n < 10®, 
taking p in [0.2, 0.5) generally leads to better performance than taking the crit- 
ical p = 0.5. There is not really a contradiction with the previous results as 
for such p, the exponent 7 is so small than there is no great difference between 
logn and n'^ . For n < 10®, the polynomial regret will appear for smaller values 
of p (i.e. p « 0.1 in our experiments). 

The numerical simulations exhibit two different types of bandit problems: 
in simple bandit problems (which contain the case where the optimal arm is a 
Dirac distribution, or the case when the smallest reward that the optimal arm 
can give is greater than the largest reward than the other arms can give), the 
performance of UCB policies is all the better as the exploration is reduced, that 
is the expected regret is an increasing function of the exploration parameter. In 
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difficult bandit problems (which contain in particular the case when the smaller 
reward that the optimal arm is smaller than the mean of the second best arm), 
there is a real trade-off between exploration and exploitation: the expected 
regret of UCBl(p) decreases with p for small p and then increases. Both types 
of problems are illustrated in Figure 5. 

6.2. The gain of knowing the horizon 

There is consistently a slight gain in using UCB-H(p) instead of UCBl(p) both 
in terms of expected regret (see Figures 4 and 5) and in terms of deviations (see 
Figures 6 to 13 in pages 20 to 22). 

The latter figures also show the following. If the agent's target is not to 
minimize its expected regret, but to minimize a quantile function at a given 
confidence level, increasing the exploration parameter p (for instance taking 
p = 0.5 instead of p = 0.2) can lead to a large improvement in difficult bandit 
problems, but also a large decrement simple bandit problems. Besides, for 
large values of p or for simple bandit problems, uCBl(p) and UCB-h(/9) behave 
similarly and thus, there is not much gain in using ucb-h policies instead of 
UCBl policies. 

6.3. The gain of knowing the mean reward of the optimal arm 

When the mean reward p* of the optimal arm is known, there is a strong 
gain in using this information to design the policy. In all our experiments 
comparing the expected regret of policies, summarized in Figures 4 and 5, the 
parameter-free and anytime policy GCL* performs a lot better than UCBl(p) and 
UCB-h(p), even for the best p, except in one simulation (for n = 100, and K = 2 
arms: a Bernoulli distribution of parameter 0.6 and a Dirac distribution at 0.5). 
In terms of thinness of the tail distribution of the regret, GCL* outperforms all 
policies in simple bandit problems, while in difficult bandit problems, it generally 
outperforms ucb1(0.2) and ucb-h(0.2) and performs similarly to ucb1(0.5) and 
UCB-h(0.5) (see Figures 6 to 13 in pages 20 to 22). 

The gain of knowing p* is more important than the gain of knowing the 
horizon. It is not clear to us that we can have a significant gain in knowing 
both p* and the horizon n compared to just knowing p* . 

7. Proofs 

7. 1 . Proof of Proposition 2. 1 

f-T ^ f-Tl: When a policy is /-T, by a union bound, the event 

,logn 



6= 3A:e{l,...,i^}, n{n)>C- 



^l 



occurs with probability at most 77^. Introduce Sk,s = 'Ylil^ii-^k.t ~ Mfc)- Since 
we have 

n K K 

^^h^Ti^it) = ^Sk,TUn) +^Tk{n)pk, 
t=l fc=l fc=l 
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Figure 4: Expected regret of uCBl(p), UCB-H(p) and GCL* for various bandit settings (1/2). 
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Figure 5: Expected regret of uCBl(p), UCB-H(p) and GCL* for various bandit settings (2/2) 
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Figure 6: Comparison of policies for n = 100 and K = 2 arms: Ber(0.6) and Ber(0.5). 
smoothed probability mass function. Center and right: tail distribution of the regret. 
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Figure 7: Comparison of policies for n = 1000 and K = 2 arms: Ber{0.6) and Ber(0.5). 
smoothed probability mass function. Center and right: tail distribution of the regret. 
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Figure 8: Comparison of policies for n = 100 and K = 2 arms: Ber(0.6) and Dirac(0.5). 
smoothed probability mass function. Center and right: tail distribution of the regret. 



Left: 











— 


CL' 


\ft 


— 


cBi(n.2) 


Ik 


— 


CB-H<0.2) 






CB1((J.',) 


l\^ 











I 


— i:cBl(0.2) 

UCB-H(n.2) 

— ucBlcn,5) 

UCB-H(0.5) 


I 






ion 1211 un iw 




Figure 9: Comparison of policies for n = 1000 and K = 2 arms: Ber(0.6) and Dirac(0.5). Left: 
smoothed probability mass function. Center and right: tail distribution of the regret. 
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Figure 10: Comparison of policies for n = 1000 and K = 2 arms: Dirac(0.6) and Ber(0.5). 
Left: smoothed probability mass function. Right: tail distribution of the regret. In this simple 
bandit problem, UCBI and UCB-H curves are almost identical. 
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Figure 11: Comparison of policies for n = 1000 and K = 2 arms: Unif([0.5,0.7]) and 
Unif([0.4,0.6]). Left: smoothed probability distribution function. Right: tail distribution 
of the regret. In this simple bandit problem, UCBI and uCB-H curves are almost identical. 
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Figure 12: Comparison of policies for n = 1000 and K = 3 arms: Ber(0.6), Ber(0.5) and 
Ber(0.5). Left: smoothed probability mass function. Center and right: tail distribution of the 
regret. 
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Figure 13: Comparison of policies for n = 1000 and K = 5 arms: Ber(0.7), Ber(0.6), Ber(0.5), 
Ber(0.4) and Ber(0.3). Left: smootiied probability mass function. Center and right: tail 
distribution of the regret. 



we have 

Rn = Sk*,n - 'S'fe._Tfc.(n) ~ 2^ '5'fe,Tfc(n) + 2^ AfeTfe(n). (7) 

k^k* k^k* 

LetT = Y.k^k' Tk{n) = n^Tk^{n),t* = Y^k^k^ C'-^, andiy = niaxo<s<t.(5fc.,„- 
Sk*,n-s)- Since Sk*,n — Sk-'.Tk-{n) < W^ on the complement S,i of ^i, we have 



k^k' k^k' 



(8) 



Consider the events 



^2 = \w>Y. 

I k=jtk* 



ICIilogn 



^3,k 



and 



max hSk^s)> \H^-;— 
l<s< ^'°|" V 2 Afc 



A: /:/*:* 



From HocfFding's maximal inequality, we have 

\2\ 



(6) < exp I v^ "^ .^i„„^N/A2 I < exp ( - /Jlogn) = — < 



Ek^kAC\ogn)/Al J- ^' ^ ^' nP-f{n)- 
We also use Hoeffding's maximal inequality to control Ve{£,3,k)- 



Pe(C3,fc) < exp 



(Clogn)/A2 



1 a 
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By gathering the previous results using a union bound, we have P(^) < ^ ^^F . 
Besides on the coniplcmcnt of ^, by using (8), we have 

£, ^ >r- fcpiogn .(-^ fcpiogn ,-^ Clogn 



We have thus proved that 



V0 e e,Vn > 1, Vg (r„ > (^+726^)^") 



log n\ C + 2a 



hence the policy is f-TZ. 

f-wT ^ f-wTZ: it is exactly the same proof as for f-T => f-TZ since the core 
of the argument is independent of the position of "V6'" with respect to "3C, C" . 

f-wTZ => f-wT: let us prove the contrapositive. So we assume 



30 e e such that A 7^ 0, VC", C" > 0, 3n > 1, 3k ^ k* , ¥g [ Tk{n) > C'-^ > .. ^ • 



,,logn^ ^ C 



(9) 
It is enough to prove that for this 9, we have 

VC > 9K/A, yC > a, 3n > l.Vg ( i?„ > C^ ) > -^. 

To achieve this, we consider C" = (/3+2)C/A and C" = max (2(7, maxm<x fi'fn)) 
in (9) and let fc' 7^ k* be such that the event 

? ^ <Tk' (n) > C ^27 



holds with probability greater than C / f{n) = 2C/ f(n). From (9) and using 

c 



C" > ina.Xrn<K fim), we necessarily have n> K. Let L = log (^^-^nK^ and 



r = I Vfc 7^ fc*, Vs e {1, . . . ,n}, |5fc,,| < -^^ i n I ^'^ ^ {!' • • • '"}' l^fc%n-'5fc%n-.| < 

By Hoeffding's inequality and a union bound, this event holds with probability 
at least 1 — C/f{n). As a consequence, we have P(^' n S,") > C/f{n). We now 
prove that on the event ^' n ^", we have 

Rn ^ C — ; — • 

A 

First note that for any a > the function s 1^ as — \/2sL is decreasing on 
[0, 2^1 and increasing on [w^, +00), and that 



„logn CL 

Ai - A?, ^ 



T.^(n)>C'^>^2 
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since ^nK < ^n^ = n^^'^ < n^'/C. Then, by using (7) and Tfe. (n) 
" ~ J2k^k* Tk{n), we have 

Rn > ~\Sk-',n - 'S'fc*,Tfc.(n)l - 2^ \Sk,Tk(n)\ + /^ ^kTk{n) 

k^k' k^k' 



' k^k* k^k' 

> Y, (AfeTfc(n)- V2Tfc(n)ij 
k^k* \ J 



k=t 

Ak'Tk'in) fAk'Tk'in) 



>^^^^^^+P^P^-V^^Z^VW + E -in A,.-V2i: 



2 / ^--' s>i 

k^k'.k^k' ^ 



,logn /C y— \ £ ^ J^ 

2Afc' I2 ^ JAfc, _,4-„2Afe 



,logn C_Z^ - — > r'^J^^ > /^ 1°S" 

- 2Afc, 6 Afe, 2A - 2Afe. " A ' 

which ends the proof of the contrapositive. 

7.2. Proof of Theorem 3.3 

Let us first notice that we can remove the A^ denominator in the the def- 
inition of /-wT without loss of generaUty. This would not be possible for the 
f-T definition owing to the different position of "V6'" with respect to "3C, C" . 
Thus, a policy is f-wT if and only if 

C 
ye ee such that A^0,3C,C > O, Vn > 2, Vfc 7^ k*, Vg (Tkin) > Clogn) < -—-. 

fin) 

Let us assume that the policy has the /-upper tailed property in 6, i.e., there 
exists C,C > 

V7V > 2, V^ ^ fc, ¥g{T,{N) > ClogN) < -^. (10) 

Let us show that this implies that the policy cannot have also the /-upper tailed 
property in 9. To prove the latter, it is enough to show that for any C", C > 

3n > 2, Vg{Tk{n) > Clogn) > -^. (11) 

since k is suboptimal in environment 9. Note that proving (11) for C = C 
is sufficient. Indeed if (11) holds for C = C, it a fortiori holds for C" < C. 
Besides, when C" > C, (10) holds for C replaced by C", and we are thus brought 
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back to the situation when C = C . So we only need to lower bound ¥g(Tk{n) > 
C log n) . 

From Lemma 3.2, ¥g{^{Xi,i) > O) > is equivalent to ¥e{f^^{Xe,i) > 
O) > 0. By independence of Xi^i, . . . ,Xk,i under ¥g, condition (c) in the 
theorem may be written as 

.(ng(-,,..>o)>o. 

Since {n,^fe f^(^.a) > o} = U„>2 {U,^, itS^Li) > i}, this readily im- 
plies that 

3r/e(0,l), pJ []|^(Xo)>,7J >0. 

Leta = P,(n,^,f^(Xo)>r;). 

Let us take n large enough such that N = [4ClognJ satisfies N < n, 
ClogN < ^ and f{n)r]*{a* - ^^^) > C" for t = [C\ogN\. For any C", 
such a n does exist since / >-+oo log" for any a > 0. 

The idea is that if until round N, arms £ ^ k have a behaviour that is 
typical of 9, then the arm k (which is suboptimal in 6) may be pulled about 
Clogn times at round N. Precisely, we prove that V^ ^ fc, ¥g(Tg{N) > 

Clog at) < j^ implies Pg(Tfe(n) > C'logn) > -S^. Let us denote At = 

nis=i...t \ Yii^k 3ir(^^.s) ^ *? f • By independence and by definition of a, we have 
¥g{At) = a*. We also have 

Vg{Tk(n)>C\ogn) > p/Tfc(iV) > ^"j 

.(qH.S}) 



> 



> 



> 



^eUtf^l f]h,iN)<C\ogN 



l=tk 

Introduce Bjsi = H^aife {^^(^) < C'log A^}, and the function q such that 

lAtHBjv — Q{{Xe^s)i^k, s=l..t, {Xk,s)s=l..NJ- 

Since Vk = i^k, by definition of At and by standard properties of density functions 
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3|, we have 



Pj^tni f] {Te{N)<C log N}\] 
= q{{xi,s)e^k, s=i..t,{^k,s)s=i..N) Yl dvi{xi^s) W dvk{xk,s) 

•^ i^k s=l..N 

s = l..t 

> rj* q({xe^s)i^k, s=i..t,i^k,s)s=i..N) Y\ di^ei^e.s) Yl '^^kixk,, 
= r7*PeM,n| f|{T,(iV)<Clog7V} Ij 



tjtk S = 1..N 

= l..t 



> via*- ^- --^ 

C' 
> 



fin)' 

where the one before last step rehes on a union bound with (10) and Fg{At) — a*, 
and the last inequality uses the definition of n. We have thus proved that (11) 
holds, and thus the policy cannot have the /-upper tailed property simultane- 
ously in environment 6 and 9. 

7.3. Proof of Theorem 4-1 

Let 6* be in 9. Consider the event 

^=lyke{l,...,K},Te{l,...,n},T\\Fk.T-F,J\l < ^lognj. 

From Massart's inequality (see [20]) applied nK times corresponding to the 

difFerent times and arms and a union bound to combine the inequalities, we 

have 

9K 

P0(e) > 1 - nif (2e-('3+i) i°g") = 1 -. 

nP 

We show that on the event ^, inequalities Tk{n) < A °g" +1 hold for any 

"k 

k ^ k* , where Sk = minggg, \\Fi,^ — i^y^^Hoo- Note that Sk > 0: if not, it would 
mean that k is suboptimal in 6 and optimal in an other environment 6, with 
Vk ~ I'k- In this case, by hypothesis there exists (. ^ k such that ^{X^^i) — 

Pg-a.s. Thus is almost surely removed during the first rounds of the policy 
and, as 9 is finite, all of these problematic 9 are removed almost surely. Note 
also that 9 cannot be removed: it is readily seen that Vg I ^{Xi^i) > 0) = 1 

for all ^ e 9 and, still because 9 is finite, it is almost sure that JI0-{Xi_i) > 
for all G 9. A last consequence of the finiteness of 9 is that terms 5k are 
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uniformly bounded away from zero over 0, and so are the terms A^, so that 
the incquahties we are going to prove easily lead to the conclusion of the proof. 

Assume by contradiction that there exists k ^ k* such that T^.{n) > 2 °^" ' 

°k 

1. Then there exists t < n such that /* == A: and rfe(t - 1) > ?(^±l|MlI. 

k 

As arm k is chosen at round t, we have: 

n,{t-i) M \\h',T^it-i)-F,^4i > n{t-\) mf iiA,T,(t-i) -i^.jiL 

On the one hand, we have: 

^ logn > Tfc.(i - 1) Jnf |lFfc.,T;(t-i) - F^,, \\1, 
and on the other hand 



^n{t-\) mf ||Ffc.T,(t-i) - F^Aoo > VTk{t-l) (Sk - \\Fk.n{t-i) - Fu. 



By combining the former inequalities, we get: 

y^logn > VTkit-l)5, - y^logn 
and 



2{(3 + l)\ogn 
SI ' 

which is the contradiction expected. 



Tkit-1)< ^2 



7.4- Proof of Theorem 4-2 

The proof of the first part of the theorem is the same as the previous section 
7.3, except that one has to substitute Sk by dk and that the dk {k ^ k*) are not 
necessarily non negative. Indeed, the distance \\Fk^T — F^J\oo equals \Xk,T — f^k\ 
in the context of Bernoulli laws. 

The proof of the second part is similar to the one of Theorem 3.3: we assume 
by contradiction that there exists a policy such that 

3C,C > 0,ye e e, Vn > 2, V/c ^ r, Pe {Tk{n) > Clogn) < ^ 
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The main difference is that we cannot fix 6, 9 such that 6* € O^, 6' G 8 \ 0fc 
and /ifc = jJLk- The hypothesis only allows us to take ^k and fik arbitrarily close. 
This means that we are allowed to consider two sequences (6'")„>i and (6'")„>i 
such that, for all n > 1 (with obvious notations): 

. 0" e Gfc, ^" e e \ Gfc, 

where N — \AC log n\ . 

On the other hand, the hypothesis readily implies that 

V0,0'ee, V^e{l,...,if}, fi{i) = h>^ 

duf, in 

and 

e^k l^k 

Let us denote a = 7^-^ and At = flLi llli^^fc 5i^(^^,s) ^ "}• ^y indepen- 
dence, we have Pg{At) — a*. 

To find a contradiction, we set t — [ClogiVj and we adapt the reasoning of 
the former proof. 
If n is chosen large enough, one has N < n and C log N < jj^, and then: 



N 
2K 



V iTk{n) > Clogn) > Pg„ [ TkiN) > ^ 



> P.„ I fl T,{N) < 



> 



.e^k 
f]{T,iN)<C\ogN} 



.e^k 



> 



An I f|{r,(7V)<ciogiV} 

[i^k 

Let us denote _Bjv — Cle^k {^^(^) < Clog A^}. Bn is measurable w.r.t. Xk.i, ■ ■ ■ ,Xk^N 
and Xi^i, . . . , Xi^t (^ 7^ k), and At is measurable w.r.t. Xi,i, . . . , Xi,t (^ 7^ k), 
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so that wc can write 

lAtnBjv — Ct,N {{Xi^s)i^k, s=l..t, {Xk,s)s=l..N) ■ 

By properties of iyj} and vj^ and by definition of At we have 



> 



Pg. Utn<^ f]{T,{N) <C log N} 

j Ct.N {{xts)l^k, s = l..t,{xk.s)s=l..N) \\ '^^i(^^^s) II dv^{xk^s) 
'' e^k s=l..N 

S = l.-t 

Ct.Ni{xe,s)i^k, s = i..tAxk,s)s=i..N)a* Y[ di'"ixi,s) Yi ['^^'^'^'^ki^k.s) 

•^ /) ^ 7^ ^ — 1 AT 



i ^ k S^L.N 

■■ = l-.t 



= |-Pfl. Ut n m {T,{N) < ClogN} 

- 2 y" f{N) ) ■ 

By straightforward calculations, one can then show that /(n)Ps„ {T}~{n) > Clogn) > 

+00, which is the contradiction expected. 

7.5. Proof of Theorem 4-3 

The proof is similar the one of Theorem 4.1, except that we use Hoeffding's 
inequality rather than Massart's one. Consider the event 

^^ hke{l,...,K},se {l,...,n},s{Xk,s-f^k)^ < -^—\ogn 

From Hoeffding's inequality applied 2nK times corresponding to the different 
times and arms and a union bound to combine the inequalities, we have P(^) > 
1 — 2nKe'^^'^^^^ '°s" = 1 — ^. We will prove by contradiction that on the event 
C, we have Tfc(n) < 1 + 2(/3+i)^i°g» ^j. ^u k ^ k* . For this, consider k ^ k* 

such that Tk{n) > A °^" + 1- Then there exists t < n such that It — k and 

k 

Tk{t — 1) > .2 °^ " ■ Since the arm k is chosen at time t, it means that 

k 

n{t-l){^l*-Xk.T,(t-l))l < Tfe.(i- 1)(m* -Xfc.^T,.(t-i)) + . (12) 
Let us split the proof into two cases. 

First case: Xk,Tkit-i) > A^*- 
ThenXk.T,(t^l)-^^k > Afe and Tfe(t- l)(Xfe,T,(t-i) -Mfc)' ^ Tk{t-l)Al. The 
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contradiction readily comes from the definition of ^. 

Second case: X^Tkit-i) < M*- 
From inequality (12) one has -'ffe'^Tj.* (i-i) < fJ-* ^ ^nd (12) can be written as: 

Tk{t-i){Xk,n(t-i)- fj-*) <Tk*{t-i){Xk',n,it-i)- f^*) ■ 

On the one hand, we have: 
and on the other hand 



\/Tk{t-l)\Xk^Tk{t-i) - f^*\ > \/Tk{t-l) (Ak- \Xk^T^(t-i) - 



MA; 



^ ^^^i^^-{4B^ 



VTfe(t -l)Afc - y ^^^ log n. 



The former inequalities leads to 



+ 1 , — -, r, /,3 + l , , 2(/? + l)logn 



Thus there is a contradiction, meaning that there is no k such that Tk{n) > 

2(;3+l)logK -, 

7. 6. Proof of Theorem 5. 1 

We assume by contradiction that there exists a f-wT policy. As in the proof 
of Theorem 3.3, on can remove the A^ denominator, so that we have: 

3C,C> 0, Vn > 2, V^ ^ fc, Vg{Ti{n) > Clogn) < -^. 

Let us show that this implies that the policy cannot have also the /-upper tailed 
property in 9. To prove the latter, it is enough to show that for any C", C" > 

3n > 2, Pg (Tfe (n) > C log n) > -^, (13) 

since k is suboptimal in environment 9. 

Similarly to the proof of theorem 3.3, proving (13) for C" = C is sufficient. 
Moreover, there exists rj e (0, 1) such that the event ^ = S Ilf^i ^(Xe.i) ^ Vf 
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has probability a > under Fg. Wc denote At = 0^=1...* {llf=i ^i^e,s) > v], 
and by independence we have Vg{At) ~ a*. 

Let us set N = \KC log n] , choose n large enough so that n > N , and 
denote Y a r.v. that equals the index of an arm among those that have been 
pulled the most after time step A'', e.g. 

Y = min f argmaxjg m ;^i Ti{N) J . Obviously, such an arm has been pulled at 
least Clogn at step N (i.e. Ty{N) > Clogn a.s.), so that one has: 

Vg (Tkin) > Clogn) > Vg (T,(iV) > Clogn) > P,- [Y ^ k) > Vg {A^ n {Y ^ k}) . 

Introduce the function q such that 

lAjvn{y=fe} = q\{X£,s)l<£<K, s = 1..n)- 

One has: 

Pg (^AT n {y = fc}) = I q{{xi>^s)i<i<K, s=l..N,{Xk,s)s=l..N) W dDe{xi^s) 

1 < « < if 

s = 1..N 

> rj'^ q{ixi^s)l<e<K, s=l..N,iXk,s)s=l..N) Y\ dvfXxi^s) 



1 < e < K 

: = l..iV 



= v^Pg (An n{Y = k}) > 77^ (¥g{AN) - Pg{Y ^ k)) 

> T]^a^ - rj'^Vg (31 ^ k, Ti{N) > Clogn) 

> r]^a^ - rj'^Vg (31 ^ k, Ti{n) > Clogn) 



> {rjo) — rj 



N ^^(if-l)C 



f{n) 



As A^ is of order log ri, it is then readily seen that /(n)Pgi (Tfc(n) > Clogn) 
+00, hence the result. 
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